Skip to content

fix: make Parquet file schema type coercion work on nested schemas - #25193

Merged
adriangb merged 1 commit into
apache:mainfrom
hstack:fix-make-parquet-schema-type-coercion-nested
Sep 15, 2026
Merged

fix: make Parquet file schema type coercion work on nested schemas#25193
adriangb merged 1 commit into
apache:mainfrom
hstack:fix-make-parquet-schema-type-coercion-nested

Conversation

@adragomir

@adragomir adragomir commented Sep 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor
  • ATM, the Parquet file schema type coercion applies view transforms for example only on top-level fields, but not on nested fields.
  • For some Parquet schemas, where we request a view transform for a nested field, this is translated to an unmodified schema passed to the reader. Then, when we try to apply predicates on the actual data on that nested field, the actual predicate execution needs a cast, because the schemas are different - the predicate refers to a string view, but the actual data schema refers to a string

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

  • Modification of the apply_file_schema_type_coercions function to work on nested fields
  • Respective tests for the functionality

What is the testing strategy for this PR?

  • All tests pass
  • Newly added unit tests pass

Are there any user-facing changes?

  • Unsure ? Probably not ? The documentation in the function does NOT explicitly say the current behavior.

@github-actions github-actions Bot added the datasource Changes to the datasource crate label Sep 11, 2026
@aditanase

aditanase commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

cc @adriangb - this is closely related with the nested schema pruning work that is ongoing. We discovered this by investigating a 2x slowdown on delta-rs next vs old TableProviders. Turned out that inconistent Utf8View handling in nested schemas was triggering the more expensive Utf8 decoding unnecessary

@codecov-commenter

codecov-commenter commented Sep 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.54751% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 81.91%. Comparing base (c2cf289) to head (4ba5397).
⚠️ Report is 11 commits behind head on main.

Files with missing lines Patch % Lines
...tafusion/datasource-parquet/src/schema_coercion.rs 99.54% 1 Missing ⚠️
Additional details and impacted files 
@@           Coverage Diff            @@
##             main   #25193    +/-   ##
========================================
  Coverage   81.90%   81.91%            
========================================
  Files        1134     1134            
  Lines      425261   425481   +220     
  Branches   425261   425481   +220     
========================================
+ Hits       348325   348535   +210     
- Misses      56295    56301     +6     
- Partials    20641    20645     +4

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@adriangb

adriangb commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

could some of these be SQL / SLT tests?

ci is red, it might not be this prs fault but could you rebase and re-run please?

@adragomir
adragomir force-pushed the fix-make-parquet-schema-type-coercion-nested branch 2 times, most recently from 96d144b to d93b5b9 Compare September 14, 2026 07:32
@adragomir
fix: make Parquet file schema type coercion work on nested schemas
4ba5397 
- ATM, the Parquet file schema type coercion applies view transforms for example only on top-level fields, but not on nested fields. 
- For some Parquet schemas, where we request a view transform for a nested field, this translated to an unmodified schema passed to the reader. Then, when we try to apply predicates on the actual data, the actual predicate execution needs a cast, because the schemas are different.
@adragomir
adragomir force-pushed the fix-make-parquet-schema-type-coercion-nested branch from d93b5b9 to 4ba5397 Compare September 14, 2026 08:13
@adragomir

adragomir commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

@adriangb fixed fmt, checking rust CI build.
Unfortunately, don't know how to do this in an slt test, explain analyze output is completely unchanged, and I don't see a test where I have access to internal reader stuff to have something different in it that I can put in an slt test. I read through parquet_filter_pushdown.slt, and I can't figure out how to do it.

adriangb
adriangb approved these changes Sep 15, 2026
View reviewed changes

@adriangb adriangb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are right, I’m sorry for the unreasonable ask!

@adriangb
adriangb added this pull request to the merge queue Sep 15, 2026
Merged via the queue into apache:main with commit 3a6a18c Sep 15, 2026
41 checks passed
@adragomir
adragomir deleted the fix-make-parquet-schema-type-coercion-nested branch September 15, 2026 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adriangb adriangb adriangb approved these changes

Assignees

No one assigned

Labels

datasource Changes to the datasource crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Parquet schema file type coercion functionality is not applied on all fields in nested schemas, only on the top-level fields

4 participants

@adragomir @aditanase @codecov-commenter @adriangb